Add disposable performance benchmark#708
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #708 +/- ##
==========================================
- Coverage 70.53% 70.42% -0.11%
==========================================
Files 61 61
Lines 13546 13513 -33
==========================================
- Hits 9554 9517 -37
- Misses 3992 3996 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Using similar structure is fine, but please move it to a separate file at least in qubes.tests, so it can be scheduled separately. And likely the same for the file in tests/ too - given you added "if dispvm test do this else do the other" more or less everywhere. |
fbcbe67 to
affd60b
Compare
b49c778 to
9f939b7
Compare
This comment was marked as outdated.
This comment was marked as outdated.
|
It was worse because it was not using a preloaded....
…On Fri, Jul 25, 2025, 10:33 PM ben-grande ***@***.***> wrote:
*ben-grande* left a comment (QubesOS/qubes-core-admin#708)
<#708 (comment)>
Did a table manually with the results. 16 iterations, preload normally 2,
except on concurrent that it preloads double (4).
test run time (s) average (s)
fedora-42-xfce_dispvm 136.79583648600965 8.549739780375603
fedora-42-xfce_dispvm-gui 166.28912552101247 10.393070345063279
fedora-42-xfce_dispvm-concurrent 72.48606694900081 4.530379184312551
fedora-42-xfce_dispvm-concurrent-gui 77.78402063000249 4.861501289375155
fedora-42-xfce_dispvm-dom0 159.93910823899205 9.996194264937003
fedora-42-xfce_dispvm-dom0-gui 184.57327080400137 11.535829425250085
fedora-42-xfce_dispvm-dom0-concurrent 78.68325804101187 4.917703627563242
fedora-42-xfce_dispvm-dom0-concurrent-gui 82.98177742300322
5.186361088937701
fedora-42-xfce_dispvm-preload 77.30045937601244 4.831278711000777
fedora-42-xfce_dispvm-preload-gui 73.71754707499349 4.607346692187093
fedora-42-xfce_dispvm-preload-concurrent 57.57939739200810
3.598712337000506
fedora-42-xfce_dispvm-preload-concurrent-gui 58.38620778400218
3.649137986500136
fedora-42-xfce_dispvm-preload-dom0 165.32496509299380 10.332810318312113
fedora-42-xfce_dispvm-preload-dom0-gui 183.59772987499310
11.474858117187068
fedora-42-xfce_dispvm-preload-dom0-concurrent 82.48804083499999
5.155502552187499
fedora-42-xfce_dispvm-preload-dom0-concurrent-gui 90.14072562199726
5.633795351374829
fedora-42-xfce_vm 9.68348605500068 0.605217878437542
Interesting that the dom0 discrepancy was not as big, it was actually
worse...
—
Reply to this email directly, view it on GitHub
<#708 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BCE2O4PDT4XMP2IEDRNW5X33KKIARAVCNFSM6AAAAACCJ3DJJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMRQGI3DGMZRG4>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
2fb5fc3 to
9688a00
Compare
|
16 iterations, max preload of 2, but on concurrency it is 4.
|
|
Those numbers look much more plausible :) |
726e75f to
87085f9
Compare
Please add #712 to be OpenQAd together with this PR. |
|
Still ironing out the tests to avoid expiring the timeout. Testing with 16GB on my testbench. |
87085f9 to
b14f6f8
Compare
|
Concurrent tests (4 qubes at a time) on a 16GB system can lead to qmemman logging: Xen free = N too small to satisfy assignments!. The good news is that the tests are not failing though, hoping OpenQA likes it. |
Besides this, full test of fedora-42-xfce with 16GB RAM was completed in 58 minutes (time increased due to cool down/delay/sleep added), |
* origin/pr/712: Skip memory assignment on domain still starting Pull request description: For: QubesOS/qubes-issues#1512 --- This does not fix all cases of *Failed to set memory* (but lowers it a lot) and does fix the TypeError exception being raised when `mem_used` is `None`: #708 (comment)
The timeout is 60 seconds for a single API call, I don't understand yet why it happened. I added the logs of the previous run also for comparison, no iteration is reaching over 60 seconds. The test Adding the sleep/delay already helped no happen many times. Sleep is already a considerable amount of the test run, but I see no other alternative to reduce load. qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.disp9411:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp5696' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp8810' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.023 0.000 0.800 2.590 3.413 1% 0% 23% 76%
2 0.064 0.000 1.107 2.296 3.467 2% 0% 32% 66%
3 0.063 0.000 16.475 2.661 19.200 0% 0% 86% 14%
4 0.063 0.000 9.338 4.925 14.326 0% 0% 65% 34%
5 0.017 0.000 7.564 2.953 10.533 0% 0% 72% 28%
6 0.012 0.000 10.551 3.176 13.739 0% 0% 77% 23%
7 0.090 0.000 4.686 3.769 8.544 1% 0% 55% 44%
8 0.033 0.000 6.934 2.924 9.891 0% 0% 70% 30%
9 0.059 0.000 12.521 3.143 15.722 0% 0% 80% 20%
10 1.407 0.000 6.222 2.889 10.518 13% 0% 59% 27%
11 0.065 0.000 10.463 3.684 14.213 0% 0% 74% 26%
12 0.041 0.000 11.537 3.548 15.125 0% 0% 76% 23%
13 0.059 0.000 9.616 4.399 14.074 0% 0% 68% 31%
14 0.148 0.000 8.636 2.714 11.498 1% 0% 75% 24%
15 0.071 0.000 7.073 3.038 10.182 1% 0% 69% 30%
16 0.097 0.000 7.518 2.499 10.114 1% 0% 74% 25%
Run time (iterations=16 average=3.091): 49.463s
WARNING:vm.disp7168:Failed to set memory
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp9783:Failed to set memory
WARNING:vm.disp8679:Failed to set memory
WARNING:vm.disp8617:Failed to set memory
Traceback (most recent call last):
File "/usr/lib/qubes/tests/dispvm_perf.py", line 347, in call_api
target_qube.run_service_for_stdio(service, timeout=60)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/qubesadmin/vm/__init__.py", line 331, in run_service_for_stdio
stdouterr = p.communicate(input=input, timeout=timeout)
File "/usr/lib64/python3.13/subprocess.py", line 1222, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 2129, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 1269, in _check_timeout
raise TimeoutExpired(
...<2 lines>...
stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '['/usr/lib/qubes/qrexec-client', '-d', 'disp9783', '-T', '-w', '60', 'DEFAULT:QUBESRPC qubes.WaitForSession+ dom0']' timed out after 60 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/qubes/tests/dispvm_perf.py", line 575, in <module>
main()
~~~~^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 571, in main
run.run_test(test)
~~~~~~~~~~~~^^^^^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 518, in run_test
result = self.run_latency_api_calls(test)
File "/usr/lib/qubes/tests/dispvm_perf.py", line 401, in run_latency_api_calls
all_results = asyncio.run(self.api_thread(test, service, qube))
File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/lib64/python3.13/asyncio/base_events.py", line 725, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 385, in api_thread
all_results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/qubes/tests/dispvm_perf.py", line 355, in call_api
raise Exception(f"service '{service}' failed: timeout expired")
Exception: service 'qubes.WaitForSession' failed: timeout expired
FAILFedora
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp4014' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp5806' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp344' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp9837' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.014 0.000 1.373 3.346 4.732 0% 0% 29% 71%
2 0.055 0.000 1.056 2.444 3.555 2% 0% 30% 69%
3 0.055 0.000 10.905 3.997 14.958 0% 0% 73% 27%
4 0.055 0.000 17.128 4.277 21.460 0% 0% 80% 20%
5 0.014 0.000 10.272 5.372 15.658 0% 0% 66% 34%
6 0.017 0.000 10.376 4.837 15.230 0% 0% 68% 32%
7 0.065 0.000 6.037 2.625 8.728 1% 0% 69% 30%
8 0.122 0.000 6.722 4.096 10.939 1% 0% 61% 37%
9 0.686 0.000 10.647 3.124 14.457 5% 0% 74% 22%
10 0.087 0.000 12.985 5.142 18.213 0% 0% 71% 28%
11 0.042 0.000 17.388 3.377 20.806 0% 0% 84% 16%
12 0.070 0.000 7.463 3.381 10.914 1% 0% 68% 31%
13 0.053 0.000 10.591 2.980 13.623 0% 0% 78% 22%
14 0.119 0.000 8.432 3.004 11.555 1% 0% 73% 26%
15 0.129 0.000 10.346 2.484 12.959 1% 0% 80% 19%
16 0.071 0.000 10.126 1.700 11.898 1% 0% 85% 14%
Run time (iterations=16 average=3.525): 56.393s
WARNING:vm.disp9084:Failed to set memory
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp1791:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
iter dom disp exec clean total dom% disp% exec% clean%
1 0.020 0.000 3.453 4.598 8.070 0% 0% 43% 57%
2 0.065 0.000 3.903 4.130 8.098 1% 0% 48% 51%
3 0.064 0.000 15.426 6.935 22.425 0% 0% 69% 31%
4 0.064 0.000 16.346 5.871 22.281 0% 0% 73% 26%
5 0.072 0.000 18.769 3.118 21.960 0% 0% 85% 14%
6 0.101 0.000 11.916 4.934 16.952 1% 0% 70% 29%
7 0.083 0.000 10.631 3.205 13.919 1% 0% 76% 23%
8 0.076 0.000 6.412 4.423 10.911 1% 0% 59% 41%
9 0.091 0.000 18.597 5.466 24.154 0% 0% 77% 23%
10 0.040 0.000 14.117 6.272 20.429 0% 0% 69% 31%
11 0.056 0.000 14.380 4.445 18.881 0% 0% 76% 24%
12 0.094 0.000 18.648 6.210 24.952 0% 0% 75% 25%
13 0.116 0.000 10.686 5.547 16.349 1% 0% 65% 34%
14 0.373 0.000 6.203 4.628 11.204 3% 0% 55% 41%
15 0.023 0.000 16.141 2.457 18.620 0% 0% 87% 13%
16 0.051 0.000 10.429 2.190 12.670 0% 0% 82% 17%
Run time (iterations=16 average=4.614): 73.828s
okWhonix-Workstation
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_406_dispvm_preload_concurrent_api:starting
ERROR:appmenus:Failed to update appmenus for 'disp2840' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp7929' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.016 0.000 1.655 2.779 4.451 0% 0% 37% 62%
2 0.025 0.000 1.747 4.055 5.827 0% 0% 30% 70%
3 0.047 0.000 16.768 7.214 24.030 0% 0% 70% 30%
4 0.047 0.000 11.803 4.451 16.301 0% 0% 72% 27%
5 0.014 0.000 15.058 5.069 20.142 0% 0% 75% 25%
6 0.018 0.000 16.295 6.438 22.750 0% 0% 72% 28%
7 0.063 0.000 9.997 4.802 14.863 0% 0% 67% 32%
8 0.191 0.000 9.151 3.237 12.579 2% 0% 73% 26%
9 0.046 0.000 11.509 3.876 15.431 0% 0% 75% 25%
10 0.038 0.000 16.261 5.643 21.942 0% 0% 74% 26%
11 0.079 0.000 10.522 3.750 14.351 1% 0% 73% 26%
12 0.056 0.000 11.093 4.775 15.924 0% 0% 70% 30%
13 0.073 0.000 11.317 3.063 14.454 1% 0% 78% 21%
14 0.068 0.000 12.650 4.146 16.864 0% 0% 75% 25%
15 0.033 0.000 11.376 3.998 15.407 0% 0% 74% 26%
16 0.021 0.000 12.843 2.381 15.245 0% 0% 84% 16%
Run time (iterations=16 average=4.236): 67.782s
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp3153:Failed to set memory
iter dom disp exec clean total dom% disp% exec% clean%
1 0.019 0.000 4.108 3.652 7.778 0% 0% 53% 47%
2 0.041 0.000 4.570 3.424 8.035 1% 0% 57% 43%
3 0.041 0.000 27.668 5.829 33.538 0% 0% 82% 17%
4 0.053 0.000 19.230 6.941 26.224 0% 0% 73% 26%
5 0.043 0.000 10.327 7.314 17.684 0% 0% 58% 41%
6 0.034 0.000 16.031 5.359 21.425 0% 0% 75% 25%
7 0.150 0.000 12.879 4.751 17.781 1% 0% 72% 27%
8 0.095 0.000 5.860 6.112 12.067 1% 0% 49% 51%
9 0.132 0.000 20.925 7.305 28.362 0% 0% 74% 26%
10 0.052 0.000 12.197 6.264 18.513 0% 0% 66% 34%
11 0.099 0.000 22.760 6.863 29.722 0% 0% 77% 23%
12 0.093 0.000 11.509 4.199 15.802 1% 0% 73% 27%
13 0.063 0.000 14.292 6.674 21.030 0% 0% 68% 32%
14 0.101 0.000 14.519 4.781 19.401 1% 0% 75% 25%
15 0.062 0.000 16.909 3.194 20.165 0% 0% 84% 16%
16 0.104 0.000 14.017 2.021 16.143 1% 0% 87% 13%
Run time (iterations=16 average=5.260): 84.159s
WARNING:vm.disp7226:Failed to set memory
ok |
|
I'm not sure what happened there, the console log at https://openqa.qubes-os.org/tests/148784/file/system_tests-qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407.guest-disp9783.log doesn't show any failures... Can you include stderr of the qubes.WaitForSession service also in case of timeout? |
For: QubesOS/qubes-issues#1512
Requires: QubesOS/qubes-core-admin-client#376